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Interim  Profile 


Development  and  Trial  of  a  Method  to  Rapidly  Measure 
Software  Engineering  Maturity  Status 

Abstract:  Development  of  an  interim  profile  (IP)  method  was  driven  by  a 
business  need  to  rapidly  measure  an  organization’s  software  engineering 
process  maturity  between  organizational  software  process  assessments 
(SPAs).  This  document  provides  information  about  the  process  used  to 
develop  the  method  and  a  description  of  the  method  to  software  engineering 
process  group  (SEPG)  members  and  practitioners  responsible  for  diagnosing 
software  process  maturity.  This  document  also  addresses  the  next  steps  in  the 
further  development  and  use  of  the  interim  profile  method. 


1  Introduction 

1 .1  Need  for  the  IP  Method 

In  the  past,  senior  and  middle  managers  had  limited  capability  to  measure  the  status  of 
software  process  improvement  between  software  process  assessments  (SPAs).  A  SPA 
usually  is  done  18  to  36  months  after  an  improvement  plan  is  begun.  Following  a  SPA, 
organizations  often  lacked  a  CMM-based  method  for  tracking  progress  toward  software 
process  improvement  goals.  Software  Engineering  Institute  (SEI)  clients  voiced  a  need  to 
measure  their  progress  against  their  organizations’  CMM  objectives  on  a  more  frequent  basis. 
The  value  of  the  interim  profile  method  is  that  it  makes  it  easier  to  get  an  objective  and  reliable 
status  update  of  process  improvement  progress  on  an  annual  basis.  This  is  a  need  which  is 
not  met  by  the  current  practice  of  providing  progress  and  status  briefings  to  managers. 

1.2  Background 

In  August  1 990,  Pacific  Beil  (San  Ramon,  California)  conducted  an  SEI  self-assessment  in  the 
Systems  Technology  organization.  The  SEi-trained  assessment  team  reported  findings,  and 
a  variety  of  action  plans  were  implemented  to  address  the  issues  identified.  At  the  time,  the 
organization  consisted  of  three  software  development/maintenance  departments  totaling  over 
1800  employees.  Each  department  developed  process  improvement  action  plans  and 
established  departmental  groups  (similar  to  software  engineering  process  groups  or  SEPGs) 
to  support  its  efforts.  Each  departmental  group  developed  and  used  a  variety  of  mini¬ 
assessments.  project  evaluations,  and  project  audit  methods  to  identify  further  areas  for 
improvement  In  projects/workgroups.  The  use  of  these  differing  methods  made  it  difficult  for 
senior  management  to  gain  insight  into  the  overall  status  of  software  process  improvement 
efforts. 
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Senior  management  wanted  more  frequent  information  on  the  status  and  progress  of  software 
process  improvement  efforts  undenvay  throughout  the  organization.  This  information  is  useful 
to  managers  because  comparisons  of  current  information  with  the  baseline  established  in 
software  process  assessments  play  an  important  role  in  determining  future  process 
improvement  efforts  and  making  other  business  decisions.  During  1990-91,  several  modified 
software  process  assessments  were  tried  out  by  SEPGs  within  the  1800  person  organization. 
While  these  individual  approaches  may,  in  fact,  work  best  to  enable  individual  departments  to 
carry  out  their  own  process  improvement  changes,  the  variety  of  methods  available  did  not 
provide  management  with  a  standard  yardstick  for  measuring  process  maturity  status. 
Therefore,  the  organization  needed  a  consistent  method  to  check  the  status  of  its  process 
improvement  effort.  Furthermore,  because  the  method  would  be  used  more  frequently  than 
the  SPA,  the  cost  and  effort  would  need  to  be  substantially  less  than  a  SPA. 

Senior  management  at  the  Pacific  Bell  Systems  Technology  organization  asked  their  central 
software  engineering  process  group  to  develop  a  means  for  measuring  all  of  the  software 
projects  in  the  organization  compared  to  the  CMM  In  September  of  1991.  In  January  1992, 
Pacific  Bell  representatives  met  with  the  SEI  Empirical  Methods  (EM)  Project  to  discuss  the 
possibility  of  working  together  to  meet  the  senior  management’s  need.  As  a  result  of  those 
discussions,  a  Pacific  Bell  resident  affiliate  position  was  established  at  the  SEI.  The 
collaborative  relationship  supported  codevelopment  of  the  interim  profile  method.^ 

1.3  SEI  Involvement 

The  Pacific  Bell  partnership  presented  a  new  opportunity  for  the  SEI  Process  Program.  This 
opportunity  to  collaborate  with  an  industry  partner  provided  four  leverage  points  for  the  SEI. 
The  partnership  was  a  means  to: 

•  extend  SEI  resources, 

•  meet  customer  requirements  for  a  new  product  to  support  continuous  process 
improvement, 

•  accelerate  product  development,  and 

•  test  another  product  (the  maturity  questionnaire)  that  was  under  development. 

A  key  factor  In  the  SEI  decision  to  collaborate  was  the  invitation  by  Pacific  Bell  to  use  the 
interim  profile  development  effort  as  a  site  for  testing  early  prototypes  of  the  SEI's  CMM-based 
maturity  questionnaire.  Pacific  Bell’s  San  Ramon  facility  housed  about  2000  software 
professionals,  which  was  large  enough  to  allow  for  efficient  and  extensive  usability  testing  at 
a  single  site. 

In  return  for  Pacific  Bell’s  support,  the  SEI  assigned  members  of  its  technical  staff  to  provide 
method  design  and  testing  guidance  and  to  support  field  trials.  As  testing  was  completed,  EM 


’•This  approach  to  collaborative  product  development,  in  which  SEI's  customer  proposed  and  managed  the  devel¬ 
opment  of  a  product  was  a  first  for  the  SEI. 
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staff  assessed  the  reliability  of  product  prototypes.  The  final  role  EM  expects  to  carry  out  with 
Pacific  Bell  is  to  support  transition  of  the  method  to  the  community. 

Thus,  both  Pacific  Bell  and  the  SEI  benefited  from  the  collaborative  approach  to  product 
development. 

1 .4  Audience 

We  expect  this  report  to  interest  and  be  useful  to  two  groups:  (1 )  practitioners  who  are  tracking 
the  status  of  methods  to  use  in  determining  software  process  maturity  and  (2)  practitioners 
considering  the  option  of  collaboration  with  the  SEI  to  develop  new  methods  for  determining 
software  process  improvement  progress. 

This  report  should  assist  readers  in  understanding  what  an  interim  profile  is  and  where  it  fits 
in  an  overall  software  process  improvement  effort. 

Materials  explaining  how  to  perform  interim  profiles  will  be  forthcoming  in  1994. 

1 .5  Organization  of  This  Report 

In  Chapter  1  we  presented  a  brief  background  of  the  IP  method.  In  Chapter  2,  we  present  a 
brief  discussion  of  existing  SEI  software  process  appraisal  work  to  set  the  context  from  which 
this  work  emerged.  Some  of  the  key  features  of  the  SPA  method  are  discussed,  and  the 
difference  in  focus  between  that  method  and  interim  profiling  are  highlighted. 

Chapter  3  addresses  the  requirements,  design,  and  implementation  of  the  IP  method.  It  also 
includes  a  discussion  of  the  process  for  developing  the  final  requirements  for  the  method. 

In  Chapter  4,  the  steps  of  the  method  are  presented  following  a  graphical  depiction  of  the 
method  flow.  Finally,  Chapter  5  presents  key  findings  from  the  initial  trials  of  the  method 
followed  by  a  brief  discussion  of  future  directions  for  the  method.  This  report  is  not  intended 
to  provide  a  detailed  set  of  instructions  for  executing  the  IP  method.  Nor  does  it  include 
specific  data  from  the  participants  of  the  initial  trials. 
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2  Existing  Methods  to  Measure  Process  improvement 

2.1  SPA  Method 

The  SEI  SPA  method  is  an  intensive  5-day  investigation  by  a  team  of  6  to  10  individuals. 
During  this  time,  the  SPA  team  interviews  approximately  40  members  of  the  organization  and 
conducts  3  or  4  briefings  aimed  at  raising  awareness  and  involvement  in  an  organization’s 
software  process  improvement  effort.  Through  the  use  of  interviews  and  feedback  briefings 
process,  the  SPA  team  develops  a  deep  understanding  of  the  orgai  :ization's  software  process 
and  creates  organization-wide  buy-in  to  the  need  for  the  proposed  software  process 
improvements. 

The  SPA  objectives  are  to: 

•  learn  how  the  organization  works, 

•  identify  its  major  problems,  and 

•  engage  its  opinion  leaders  and  senior  managers  in  the  change  process. 

An  assessment  aims  to  prioritize  areas  for  improvement  and  to  provide  guidance  on  how  to 
make  those  improvements.^  After  a  SPA,  an  organization  develops  and  implements  an  action 
plan  to  address  major  findings  from  the  SPA.  Then,  after  18  to  36  months,  the  organization 
conducts  another  SPA  to  reassess  its  process  capability. 

The  IP  method’s  goal  of  tracking  an  organization’s  progress  towards  higher  levels  of  maturity 
on  a  large  scale  rapidly  and  economically  differs  from  the  goals  of  the  SPA  method.  The 
interim  profile  method  presumes  that  the  process  improvement  has  already  been  started  and 
participation  in  process  improvement  is  organization  wide.  Also,  SPAs  require  that  an 
organization  change  (unfreeze)  its  current  process  and  build  an  organizational  climate 
conducive  to  change.  For  the  IP  method,  this  change  is  underway/occurring.  IP  is  an  activity 
within  a  SPI  effort  to  provide  project  leaders  and  management  objective  data  on  an  on-going 
basis  to  track  the  improvements  already  in  progress  throughout  the  organization.^ 

2.2  The  Maturity  Questionnaire 

The  maturity  questionnaire  (MQ)  is  an  SEI  product  that  provides  initial  information  for  software 
process  appraisals.^  The  interim  profile  method  draws  heavily  upon  the  maturity 
questionnaire.  In  SPAs  and  software  capability  evaluation  (SCEs),  the  MQ  serves  as  the 
primary,  but  not  sole,  source  of  data  regarding  the  software  process  maturity  of  the  projects 
within  an  organization;  the  SPA  and  software  capability  evaluation  methods  have 


2  W.S.  Humphrey,  Managing  the  Software  Process,  Addison-Wesley,  Reading,  MA,  1989. 

^  A  table  listing  differences  between  SPA  and  IP  is  presented  on  page  13. 

The  first  version  of  the  questionnaire  (SEI  87  TR-23)  was  used  extensively  in  SPAs  and  software  capability  eval¬ 
uations  (SCEs). 
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supplemented  the  data  collected  on  the  MQ  with  inten/iews  and  document  reviews.® 
Therefore,  there  has  been  less  reliance  on  the  MQ  within  these  methods  for  determining 
software  process  maturity. 

The  latest  MQ  was  designed  to  sample  important  issues  for  software  process  improvement, 
as  described  in  the  Capability  Maturity  Model  for  Software  (CMM)  Version  1.1.  The 
questionnaire  focuses  on  a  wide  range  of  issues  covered  in  the  CMM,  with  the  expectation 
that  detailed  examination  of  specific  maturity  issues  is  accomplished  through  various 
validation  techniques  such  as  interviews,  focus  groups  and  existence  of  appropriate  artifacts. 


These  methods  are  currently  being  reviewed  and  will  use  an  upgraded  version  of  the  MQ.  A  sample  page  from 
the  IP  beta  trial  version  of  the  MQ  is  in  Appendix  A. 
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3  Overview  of  Interim  Profile  Method  Development 

Based  on  the  needs  expressed  in  Chapter  1  and  the  availability  of  draft  versions  of  the 
maturity  questionnaire  (MQ),  Pacific  Bell  and  the  SEI  began  development  of  the  interim  profile 
method.  Because  the  testing  and  delivery  of  the  IP  method  occurred  in  a  controlled  situation, 
the  SEI  was  able  to  use  this  effort  as  leverage  to  gather  usability  data  on  preliminary  drafts  of 
the  MQ.  This  resulted  in  continuous  upgrades  to  the  MQ  throughout  the  development  of  the 
IP  method. 

3.1  Requirements  Phase 

Pacific  Bell  and  the  SEI  started  gathering  extensive  requirements  for  the  method  at  the  1992 
SEPG  National  Meeting.  Meetings  were  held  with  other  companies  experienced  in  SEI  SPAs 
and  with  companies  trying  or  using  supplemental  assessment  methods  in  their  process 
improvement  efforts.  The  companies  involved  included:  Hughes,  Hewlett-Packard,  Texas 
Instruments,  Schlumberger,  Software  Productivity  Consortium,  and  IBM  Federal  Systems 
Company.  Interviews  were  conducted  to  determine  the  needs  within  Pacific  Bell,  how  the 
product  should  be  used,  and  the  potential  future  uses  of  the  method.  Interviews  were  also 
conducted  outside  Pacific  Bell  to  determine  industry  needs,  possible  markets,  and  method 
features.  The  initial  IP  requirements  document  described  final  outputs,  functional 
requirements  lists,  and  functional  diagrams  for  each  step  in  the  IP  method. 

The  requirements  for  the  IP  method  were  distributed  within  and  outside  of  Pacific  Bell  for 
comments.  Feedback  from  reviewers  was  used  to  finalize  the  method  design.  In  particular,  the 
feedback  identified  specific  method  steps  to  check  for  accuracy  and  allow  questionnaire 
respondents  to  provide  additional  input  to  revise  the  profile  results.  Feedback  on  the 
requirements  document  also  identified  major  design  issues  to  be  addressed. 

3.2  Method  Design  Phase 

Based  on  the  requirements  information  gathered.  Pacific  Bell  and  the  SEI  made  seven  major 
decisions  at  the  beginning  of  the  design  stage  for  the  IP  method.  The  method  would: 

1 .  use  the  SEI  MQ  as  the  data  gathering  vehicle,  thus  decreasing  the  likelihood 
of  major  differences  in  initial  data  about  process  maturity  for  a  SPA,  SCE,  and 
IP; 

2.  report  the  results  of  the  questionnaire  in  key  process  area  (KPA)  profiles  both 
for  individual  projects  and  for  the  organization  as  a  whole  so  that  the  projects 
received  feedback  in  addition  to  the  organization;^ 

3.  focus  on  information  to  determine  the  maturity  status  against  the  CMM; 

4.  place  minimal  demands  on  the  participants’  time; 


A  three-point  scale  of:  ‘not  satisfied,*  ‘partially  satisfied, ‘  and  lully  satisfied”  was  used  pending  release  of  the  SEI 
upgraded  rating  method  in  1994. 
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5.  seek  economical  ways  to  collect  pertinent  supplemental  data  to  help  interpret 
the  questionnaire  results; 

6.  follow  the  confidentiality  rules  of  the  SEI  SPA  method;  and 

7.  include  a  feedback  loop  to  allow  respondents  to  provide  input  for  revisions  to 
preliminary  drafts  of  the  project  profiles. 

The  requirements  process  and  these  initial  design  decisions  led  to  the  identification  of  three 
primary  areas  of  risk  to  address  in  the  detailed  design  of  the  IP  method; 

1.  accuracy  of  results, 

2.  method  reliability,  and 

3.  appropriate  use  of  results. 

These  critical  design  areas  are  discussed  in  the  three  sections  that  follow. 

3.2.1  Accuracy  of  Results 

The  accuracy  of  results  continues  to  be  a  significant  concern  because  results  are  based 
primarily  on  the  responses  to  the  maturity  questionnaire.  Unlike  SPA,  the  interim  profile 
method  does  not  use  additional  data  collection  steps,  such  as  interviews  or  focus  groups. 
However,  the  method  takes  a  different  approach  from  a  SPA  by  having  a  number  of  team 
members  fill  out  the  questionnaire  and  gathering  additional  data  on  their  confidence  in  their 
responses. 

In  interim  profile,  the  MQ  is  completed  by  a  majority  of  project  members  as  opposed  to  a  SPA, 
where  one  or  two  members  complete  it  for  the  entire  team.  All  software  managers  and 
practitioners  are  encouraged  to  complete  the  MQ.  High  rates  of  agreement  among  the  MQ 
responses  for  project  members  increases  confidence  in  the  conclusions  drawn  from  the 
responses.  In  addition,  a  feedback  step  in  the  method  is  used  to  obtain  each  participating 
project's  verification  of  the  accuracy  of  its  results.  If  an  individual  group  does  not  believe  the 
results  accurately  reflect  their  capability  maturity  status,  a  review  process  is  in  place  to  resolve 
issues  and  make  changes  as  appropriate  to  the  projecf  s  key  prpcess  area  profile. 

3.2.2  Method  Reliability 

Consistency  in  the  method  implementation  and  analyzing  the  results  are  viewed  as  key  in 
addressing  the  reliability  of  the  method.  Reliability  issues  are  addressed  through  the  design 
of  tfre  method  and  through  the  incorporation  of  quantitative  indicators.  Use  of  the  CMM  as  an 
accepted  reference  model,  against  which  previous  assessments  and  action  plans  were 
created  at  Pacific  Bell,  ensures  consistency  with  respect  to  the  use  of  the  results.  Also,  use  of 
the  maturity  questionnaire  (MQ)  provides  the  ability  to  ask  the  same  questions  of  all 
respondents. 

Another  aspect  to  increase  reliability  of  the  method  is  the  emphasis  on  a  consistent  manner  in 
conducting  on-site  sessions.  Specific  and  consistent  instructions  for  administration  of  the 
questionnaire  were  developed  to  be  used  with  each  questionnaire  response  session.  Also,  the 
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delivery  of  the  overview  training  for  the  participants,  training  for  management,  and  the  review 
process  were  documented  and  delivery  of  each  was  conducted  in  the  same  manner,  stressing 
the  same  key  points  and  providing  an  environment  open  to  questions  from  the  participants  and 
management  alike. 

The  following  section  describes  the  specific  indices  of  reliability  and  the  methods  used  to 
compute  them. 

3.2.2.1  Indicators  of  Reliability 

For  each  question  on  the  maturity  questionnaire,  respondents  were  asked  to  select  from  one 
of  four  answers:  yes,  no,  does  not  apply,  and  don’t  know.  (See  Appendix  A,  which  contains  an 
example  page  of  the  MQ.)  These  responses  are  used  to  judge  the  implementation  of  key 
practices  in  the  process  used  at  the  organization.  In  addition  to  these  four  response  options, 
the  respondents  are  asked  to  provide  a  confidence  rating  about  their  response  to  each 
question.  These  ratings  are  used  to  capture  the  respondents’  confidence  that  the  answer  they 
gave  is  accurate.  The  confidence  ratings  provided  by  respondents’  are  used  to  quantify  the 
reliability  of  the  information  regarding  implementation  of  practices  in  the  organization/  We 
quantify  the  reliability  by  using  two  indicators: 

1.  The  overall  degree  of  confidence  is  a  global  summary  of  the  percentage  of 
responses  that  were  given  with  high  confidence  by  the  respondents. 

2.  The  yes  degree  of  confidence  is  a  similar  summary  of  the  percentage  of 
responses  given  with  high  confidence,  except  that  it  reflects  only  “yes’ 
responses. 

Computing  both  of  the  reliability  indicators  involves  the  aggregation  of  responses  across  all 
questions  on  the  questionnaire,  as  well  as  across  all  respondents  within  a  group.  For  project- 
level  profiles,  all  respondents  in  the  project  are  included.  Likewise,  for  organization-level 
profiles,  all  respondents  in  the  organization  are  included.  This  aggregation  is  needed  to 
preserve  the  anonymity  of  the  respondents. 

in  addition  to  the  two  indices  of  reliability  described  above,  other  sources  of  information  are 
used  to  reflect  contextual  information  about  the  pool  of  responses  collected.  These  indices 
reflect  the  degree  to  which  the  MQ  respondents  represent  the  complete  set  of  potential 
respondents  or  project  members. 

•  The  percentage  of  group  responding  Is  simply  the  percentage  of  the  total  group 
(project  or  organization)  who  completed  questionnaires,  in  addition  to  this 
percentage,  the  number  of  people  participating  is  reported  as  a  fraction  in  order 
to  reflect  the  size  of  the  total  group  from  which  respondents  were  selected. 


In  the  social  sciences,  the  reliability  of  a  measurement  instrument  (i.e.,  a  questionnaire)  can  be  expressed  as  the 
correlation  between  the  outcomes  (score)  on  two  different  administrations  of  the  instrument.  Due  to  the  method¬ 
ological  issues  sun'ounding  the  concept  of  process  maturity,  a  strict  numerical  (correlation)  approach  was  viewed 
as  inappropriate.  As  a  result,  we  have  used  a  proxy  to  represent  the  magnitude  of  the  correlation.  We  believe  that 
the  more  confident  a  respondent  is  in  giving  a  particular  answer,  the  more  likely  he  or  she  would  be  to  give  the 
same  answer  on  different  administrations  of  the  same  instrument. 
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Using  demographic  information  about  questionnaire  respondents,  the  experience  level 
reflected  in  the  people  responding  to  the  MQ  is  also  provided.  This  is  done  so  that  projects 
can  use  the  experience  data  to  determine  how  well  the  respondents  represented  the  total 
group  in  their  level  of  experience  and  what  impact  that  may  have  had  on  their  results.  In  order 
to  understand  the  interpretation  of  experience  level,  the  reader  must  be  familiar  with  the 
concept  of  percentiles.®  The  25th  and  75th  percentiles  for  the  distribution  of  experience  across 
all  respondents  are  calculated. 

•  The  percentage  of  respondents  in  each  prq/ecf  whose  years  of  experience  is  less 
than  the  25th  percentile  for  the  population  is  reported  to  show  the  number  of 
novices  in  the  project  (relative  to  the  entire  set  of  respondents). 

•  The  percentage  of  respondents  in  each  project  whose  years  of  experience  is 
more  than  the  75th  percentile  for  the  population  is  reported  to  show  the  number 
of  experts  in  the  project  (relative  to  the  entire  set  of  respondents). 

The  above  distinction  between  novices  and  experts  is  admittedly  artificial.  However,  with 
knowledge  of  the  history  of  the  organization  as  well  as  the  experience  level  of  the  staff, 
managers  in  the  organization  can  estimate  how  well  the  people  responding  to  the  MQ 
represent  the  experience  level  of  the  group  as  a  whole. 

3.2.3  Appropriate  Use  of  Results 

The  use/misuse  of  results  is  an  important  issue  in  all  evaluative  methods.  Therefore,  in  the 
design  of  the  interim  profile  method,  management  training  on  the  use  of  the  results  was  seen 
as  an  important  step.  This  training  explains  the  treatment  of  the  data  in  deriving  the  results  in 
order  to  help  managers  make  proper  interpretations,  and  avoid  reading  more  into  the  results 
than  is  warranted. 

Training  for  managers  includes  specific  information  on  the  ways  data  should  and  should  not 
be  used.  The  topics  in  the  training  address  the  following  classes  of  issues  that  a  manager 
might  have: 

•  assessing  the  results, 

•  planning  improvement, 

•  using  the  method  with  other  metrics,  and 

•  addressing  cautions  in  using  IP. 

Finally,  in  an  effort  to  set  appropriate  expectations  for  the  results  obtained  in  the  method, 
managers  are  presented  with  information  describing  the  similarities  between  SPA  and  IP,  as 
well  as  the  differences  between  the  two  methods.  The  similarities  are: 


For  any  given  set  of  quantitative  observations,  the  observations  may  be  separated  according  to  some  commonly 
accepted  methods  for  deriving  ‘cut-scores.’  One  such  scheme  is  to  identify  the  value  below  which  25%  of  the  ob- 
sen/ations  fall,  and  the  value  above  which  25%  of  the  observations  lie.  These  two  values  are  termed  the  25th  and 
75th  percentiles,  respectively. 
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•  Both  use  the  CMM  as  a  framework  reference  standard. 

•  Both  use  the  SEI  maturity  questionnaire  and  respondent  questionnaire  (RQ). 

•  Both  tools  help  diagnose  current  status. 

Differences  between  the  two  methods  are  listed  in  Table  1.  Additionally,  interim  profile  results 
include  both  key  process  area  (KPA)  ratings  and  reliability  factors,  thereby  making  it 
necessary  for  a  project  or  organization  to  look  at  both  components  in  order  to  determine  how 
reliable  they  believe  the  information  to  be  and  to  what  extent  they  wish  to  use  the  data  to 
continue  or  change  their  process  improvement  plans.  It  was  also  felt  that  by  providing  two 
components  of  data,  it  would  tend  to  de-emphasize  the  focus  on  comparing  results  and, 
instead,  focus  on  the  action  required  to  continue  process  improvement. 


Table  1:  Differences  Between  SPA  and  IP 


Software  Process  Assessment  (SPA) 

Interim  Profile  (IP) 

Organization  uses  to  improve  software 
process 

Organization  uses  to  check  status  of  process 
improvement  efforts  between 
organizational  self-assessments 

Findings  may  include  issues  not  in  the 
CMM 

Profile  restricted  to  CMM  KPAs 

Collaborative  approach  —  members  of  the 
organization  are  on  the  assessment  team 

Expert  approach  —  two  people  (profile 
builders)  evaluate  responses  to 
questionnaire 

Catalyst  for  process  improvement 

Checkpoint  for  process  improvement 

Organizational  results  only 

Organizational  and  individual  project 
profiles 

Trained  team  verifies  and  probes  for  more 
information  and  more  details  of  answers  to 
the  questions 

To  analyze  data,  profile  builders  use  only  the 
information  provided  through  the  responses 
to  the  questionnaire  comments  and 
organizational  knowledge  and  verification 
inputs. 
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4  Detailed  Description  of  the  Interim  Profile  Method 

This  chapter  describes  the  process  activities  of  the  interim  profile  method  and  definitions  of 
the  roles/functions  of  the  method.  The  method  is  presented  first  in  order  to  explain  the 
activities;  then,  the  roles  are  defined  to  support  the  method.  This  chapter  also  includes  a 
summary  of  the  effort  required  to  implement  each  phase  at  the  trial  sites.  Detailed 
development  and  trial  effort  data  for  Pacific  Bell  and  the  SEI  are  not  included  in  this  report. 

Figure  1  illustrates  the  general  process  activity  in  each  phase  of  the  interim  profile  method. 
These  activities  are  then  described  in  Sections  4. 1-4.4. 


Logistics 

and 

Setup 

__  J 

Initial  Data 
Collection  and 
Analysis 


Review  &  Revision 
of  Draft  Project 
Profiles 


Distribution  of  Final 
Profiles  &  Audit  of 
the  IP  Process  . 


Figure  1:  Interim  Profile  Process  Flow 


The  roles  defined  include: 

•  on-site  coordinator 

•  management  overview  trainer 

•  questionnaire  facilitator 

•  data  reduction  performer 

•  profile  builder 

•  results  distributor,  and 

•  results  reviewer. 
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4.1  Logistics  and  Setup 

During  this  phase  of  the  process  three  main  activities  occur: 

•  identification  of  the  population. 

•  preparation  of  materials,  and 

•  scheduling  and  logistics  coordination. 

Identification  of  the  population  entails  creating  a  complete  list  of  projects  within  the 
organization  requesting  the  interim  profile.  This  list  should  include  at  a  minimum: 
project/Vvorkgroup  name,  project  manager/group  leader,  and  number  of  project  members.  This 
information  is  used  for  all  communications  with  the  groups  of  respondents  and  provides 
background  information  which  may  be  of  use  when  the  questionnaire  responses  are 
summarized  and  rated. 

Preparation  of  materials  should  occur  well  in  advance  of  the  planned  administration  of  the 
questionnaire.  Questionnaires  should  be  numbered  for  tracldng  purposes.  The  use  of 
identification  numbers  stamped  on  each  questionnaire  allows  anonymous  reporting  of  the 
information,  thus  avoiding  attribution  of  information  to  individual  respondents. 

Scheduling  and  logistics  require  advance  planning  and  consideration  for  the  potential 
participants.  Multiple  two-hour  sessions  should  be  set  up  to  make  completion  of  the 
questionnaire  as  convenient  as  possible,  thereby  meeting  the  requirement  that  this  method 
cause  the  least  dismption  possible  to  the  workforce.  Communication  regarding  dates  and 
locations  should  be  made  well  in  advance  to  enable  people  to  work  this  activity  into  their 
schedule.  Alternative  appointments  should  be  made  available  for  cases  where  the  group 
cannot  be  appropriately  represented. 

Rooms  for  the  questionnaire  administration  must  be  sufficient  in  size  to  provide  comfortable 
seating  for  the  respondents.  Tables  able  to  accommodate  people  with  multiple,  spread-out 
forms  are  needed. 

Sufficient  numbers  of  questionnaire  packages,  each  containing  a  maturity  questionnaire, 
respondent  data  questionnaire,  confidence  data  questionnaire,  and  instructions,  need  to  be 
printed  and  delivered  to  the  site  in  advance. 

During  the  initial  planning  phases,  it  is  also  advisable  to  begin  identifying  potentially 
problematic  terms  used  on  the  MQ.  A  review  of  the  questionnaire  should  be  conducted  by  the 
organization  in  order  to  provide  an  explanation  of  the  language  used  in  the  MQ  from  the  CMM 
in  terms  of  the  language  that  is  used  by  the  people  who  will  be  completing  the  MQ. 

4.2  Initial  Data  Collection  and  Analysis 

Scheduling  and  logistics  coordination  requires  effective  contingency  planning  and 
coordination  with  both  the  support  staff  and  the  interim  profile  participants.  This  method 
requires  that  the  questionnaires  be  administered  in  a  group  setting,  using  standardized 
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instructions  to  ensure  that  all  respondents  have  the  same  understanding  of  the  purpose  and 
intent  of  the  method.  It  is  very  important  that  the  administrator  of  these  sessions  be 
knowledgable  in  the  st'  ject  matter  and  the  organization  in  which  the  respondents  work.  The 
sessions  typically  take  two  hours,  with  the  first  30  minutes  reserved  for  an  overview  of  the 
purpose,  scope,  outputs,  and  use  of  the  outputs. 

A  trained  facilitator  covers  issues  including  background  of  the  method  and  samples  of  profiles. 
The  facilitator  also  instructs  participants  on  how  to  complete  the  questionnaires  and  how  to 
ask  questions,  and  emphasizes  the  need  for  individual  answers,  not  consensus  of  the 
workgroup  or  others  sitting  at  the  table.  Before  the  participants  leave,  the  facilitator  checks 
the  completed  questionnaires  to  ensure  proper  workgroup  identification. 

Data  entry  for  this  type  of  effort  can  require  a  great  deal  of  time  and  effort.  It  is  very  important 
that  the  files  created  be  checked  for  accuracy  and  completeness. 

Once  the  questionnaire  data  have  been  entered  and  checked  for  accuracy,  the  first  draft  of 
the  project  profiles  is  created.  Because  of  the  need  to  examine  numeric  results  (e.g.,  number 
of  people  answering  yes  or  no)  as  well  as  comments  written  to  clarify  the  answers,  this  part  of 
the  method  is  very  human-intensive.  The  interpretation  of  comments  and  the  profile  builders’ 
knowledge  of  the  organization  contributes  a  great  deal  to  the  interpretation  of  the  responses. 
Rating  rules  were  developed  for  the  IP  method  based  on  information  gathered  during  require¬ 
ments  elicitation  from  individuals  conducting  various  versions  of  CMM-based  evaluations.^ 

4.3  Review  and  Revision  of  Draft  Project  Profiles 

The  review  cycle  is  an  extremely  important  process  in  the  interim  profile  method.  This  process 
acts  as  the  validation  step  usually  found  in  traditional  CMM-based  assessments.  Participating 
projects  must  review  their  results  before  they  are  summarized  at  the  organizational  level  and 
given  to  senior  management.  Project  results  are  confidential  to  the  project  and  are  distributed 
only  to  the  designated  project  manager  to  review  with  his  or  her  project.  A  documented  review 
process  is  included  with  the  results  so  that  teams  know  what  they  are  expected  to  do. 

After  the  project  team  receives  the  results,  all  members  of  the  team  review  the  results.  If  the 
team  has  any  questions  or  would  like  to  challenge  the  results,  they  contact  the  results 
reviewer.  The  results  reviewer  (see  Section  4.5)  will  meet  with  the  team  to  explain  the  results 
and  to  determine  if  changes  should  be  made  to  the  profile.  An  example  of  the  organization 
profile  is  presented  in  Appendix  B. 

Resolution  of  the  challenges  to  the  results  must  be  completed  within  a  designated  time  frame 
(generally  two  to  three  weeks).  Changes  to  a  profile  require  objective  evidence  to  substantiate 
the  change.  If  evidence  exists  that  contradicts  the  initial  profiles,  appropriate  changes  are 
made  to  the  project  profiles. 


The  rating  rules  will  be  rewritten  to  incorporate  the  upcoming  SEI  common  rating  framework  document.  For  that 
reason  and  others,  the  rating  ruies  used  in  the  method  trials  at  Pacific  Beil  and  other  locations  are  not  presented 
here. 
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4.4  Distribution  of  Finai  Profiies  and  Audit  of  the  IP  Process 

Using  approved  project  profile  data  from  projects,  an  organizational  profile  is  constructed. 
This  organizational  profile  is  designed  to  provide  information  to  senior  management.  The 
profile  summarizes  the  percentage  of  projects  rated  in  each  KPA  as  fully  satisfied,  partially 
satisfied,  or  not  satisfied.  The  organizational  profile  also  includes  reliability  factors 
summarized  at  the  organizational  level  as  discussed  in  Section  3.2.2.  An  example  of  the 
organization  profile  is  presented  in  Appendix  C. 

Organizational  profiles  are  available  to  all  members  of  the  organization.  They  are  distributed 
to  the  senior  management  team  in  advance  of  being  distributed  to  the  entire  organization.  On 
the  other  hand,  project  profiles  are  confidential.  Finalized  project  profiles  are  distributed  only 
to  the  designated  project  managers.  The  project  team  must  determine  if  and  how  they  will 
share  their  results  with  their  middle  and  senior  managers. 

The  interim  profile  process  should  be  evaluated  with  respect  to  its  usefulness  to  the 
organization  and  the  accuracy  of  the  results,  in  addition,  other  data  should  be  collected  to 
capture  respondents’  views  regarding  things  that  worked  well  and  things  that  did  not  work  well 
in  the  process.  The  findings  of  this  audit  process,  and  the  actions  taken  in  response  to  them, 
should  be  communicated  back  to  the  participants.  One  method  for  collecting  this  type  of 
feedback  —  the  interim  profile  results  survey  —  is  discussed  in  Appendix  D. 

4.5  Roles 

It  is  important  to  identify  roles  and  responsibilities  involved  in  the  implementation  of  the  IP 
method.  The  roles  and  their  functions  are  listed  below.  In  some  cases,  multiple  roles  may  be 
performed  by  one  individual.  This  is  dependent  on  the  organization’s  size,  culture,  experience, 
etc.,  and  is  determined  during  implementation  planning. 

On-site  coordinator 

This  role  performs  the  up  front  project  planning  on  all  aspects  of  implementation.  This 
coordinator  has  overall  responsibility  of  on'Slte  communication,  logistics  and  setup, 
sponsorship  building,  and  customization  of  the  results  distribution  and  review  process.  In 
addition,  the  on-site  coordinator  works  with  the  on-site  software  process  improvement  groups 
to  coordinate  support  and  involvement. 

Management  overview  trainer 

This  role  provides  the  management  training  to  the  appropriate/identified  senior  and  middle 
managers.  The  function  could  be  performed  by  the  on-site  coordinator,  a  sponsoring  senior  or 
middle  manager,  or  a  member  of  the  software  engineering  process  group  (SEPG).  At  Pacific 
Bell,  a  train-the-trainer  approach  was  selected.  Three  senior  managers  were  trained,  and  they 
trained  the  middle  managers. 
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Questionnaire  faciiitator 


This  role  is  responsible  for  delivering  the  up-front  overview  of  the  method  and  detailed 
instructions  on  completing  the  questionnaire  to  the  questionnaire  respondents/participants. 
The  facilitator  must  be  able  to  answer  all  questions  from  the  participants,  maintain  “control”  of 
the  session  and  ensure  that  all  questionnaires  have  been  correctly  labeled  with  project  and 
individual  identities  mapping  to  the  correct  profiles.  In  addition,  the  facilitator  is  responsible  for 
building  the  organizational  master  mapping  list  and  providing  it  to  the  data  reducer  so  that  the 
correct  questionnaire  responses  are  mapped  to  the  correct  identified  projects  and 
organizations.  This  master  mapping  list  indudes  the  organizations,  projects  in  the 
organization,  and  completed  questionnaires  in  each  project.  The  list  also  includes  total 
number  of  members  of  each  team  so  the  project  participation  data  can  be  determined. 

Data  reducer 

This  role  inputs  the  data  and  summarizes  the  responses  to  the  groupings  identified  in  the 
master  mapping  list.  A  response  report  is  generated  for  each  project/group  identified  with  the 
summarized  responses,  comments,  and  confidence  rating  data. 

Profile  builder 

This  role  is  responsible  for  producing  the  final  results  upon  completion  of  the  review  phase. 
Producing  the  results  requires  the  evaluation  and  rating  of  the  MQ  responses  using  the  rating 
rules.  This  role  requires  individuals  well  experienced  with  the  SEI  CMM,  preferably  with 
assessment  training  and  experience.  In  the  trials  conducted  to  date,  the  maximum  number  of 
profile  builders  was  three. 

Results  distributor 

In  planning  activities,  the  process  for  results  distribution  is  determined  for  both  initial  and  final 
profiles.  The  results  distributor  is  responsible  for  seeing  that  the  process  is  followed  and  that 
individual  profile  results  are  distributed  to  the  participating  project  following  the  confidentiality 
rules  discussed  previously. 

Results  reviewer 

This  role  is  responsible  for  making  decisions  on  whether  the  profile  should  be  changed  based 
on  the  information  provided  by  the  project.  Once  the  reviews  are  completed,  this  function  is 
responsible  for  notifying  the  profile  builder  and  results  distributor  of  the  changes.  This  may  be 
one  or  many  individuals. 
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4.6  Effort  Required  to  Implement  IP 

In  addition  to  the  trial  and  implementation  at  Pacific  Bell,  additional  trials  were  conducted  with 
three  industry  organizations  and  two  Department  of  Defense  software  development  groups. 
In  total,  1 1  sites  and  165  projects  have  completed  an  interim  profile. 

Table  2  depicts  the  average  or  range  of  time  to  complete  the  activity  phases  of  interim  profile 
based  on  the  work  at  Pacific  Beil  and  participating  trial  sites.  Ranges  are  provided  because 
the  effort  varied  depending  on  the  size  of  the  site,  quality  of  the  data,  and  communication 
requirements. 


Table  2:  Time  Required  to  Complete  IP  Phases 


Phase 


Description 


Completion  time 


Logistics  and  Set-Up 


Initial  data  collection  and 
analysis 


Management  training 


Participant  briefing  and 
question  completion 
(per  participant) 


Questionnaire 
administration 
(per  organization) 


Data  reduction  and  report 
generation 


Profile  analysis  and 
generation 


Distribution 


Review  and  revision 


Project  teams 


Results  reviewer 


Distribution  of 
organizational  results 


It  appears  from  the  trials  to  date  that  the  interim  profile  cost  is  approximately  one-fourth  the 
cost  of  a  full  software  process  assessment.  However,  it  is  important  to  point  out  again  that 
interim  profile  cannot  replace  a  SPA  as  the  initial  appraisal  for  launching  a  software  process 
improvement  initiative. 


4-20  hours 


1-2  days 
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5  Key  Findings 


Pacific  Bell  used  two  types  of  data  gathering  techniques  to  evaluate  the  method  and  look  for 
ways  to  improve  the  interim  profile  process.  First,  feedback  surveys  were  sent  to  all 
participating  project  managers  to  check  for  perceived  accuracy  of  the  profiles,  ability  of  the 
team  participants  to  represent  the  group,  and  finally,  whether  the  questionnaire  addressed  the 
software  engineering  processes  of  the  group.  Second,  separate  focus  groups  for  middle  and 
senior  managers,  project  managers,  and  technical  professionals  were  held  to  identify 
strengths  and  weaknesses  of  the  method  along  with  recommendations  for  improvement. 

In  general,  the  combination  of  this  data  allowed  us  to  identify  the  following  benefits  and  areas 
for  improvement  of  the  IP  method  as  described  below. 

Benefits:  the  IP  method  provides: 

•  Overall  view  of  organizational  position. 

•  Objective  view  of  individual  project  status. 

•  General  perception  of  no  penalty  among  participants,  as  a  result  of  confidentiality 
principles. 

•  Vehicle  for  more  education  and  information  on  CMM  and  process  improvement. 

Areas  for  improvement: 

•  Questionnaire  language.  The  focus  group  feedback  indicated  that  many 
participants  did  not  like  the  language  of  the  questionnaire.  They  found  it 
cumbersome  and  hard  to  translate  into  their  local  terms. 

•  Questionnaire  response  categories.  It  was  felt  that  the  response  categories 
limited  the  ability  of  the  respondents  to  identify  what  was  actually  in  place,  in 
progress.  Suggestions  were  made  to  use  a  different  kind  of  response 
classification,  ranging  from  “Not  done  at  all,”  to  partially  in  place,  to  fully 
operational. 

•  Rating  rules.  This  issue  is  directly  related  to  the  response  categories. 

Participants  felt  that  the  criteria  for  answering  yes  was  unreasonable  and  limiting. 

Again,  it  was  felt  that  a  different  response  category  could  improve  this. 

•  More  coordination  of  on-site  process  improvement  support.  Project  teams 
expressed  the  need  for  more  support  from  either  the  local  SEPG  or  core  software 
engineering  process  improvement  (SEPI)  team  to  help  them  understand  the 
results  and  use  them  as  a  basis  for  improvement  action.  This  particular  issue  has 
been  addressed  in  interim  profile  trials  at  subsequent  non-Pacific  Bell  sites. 

To  date,  the  method  has  been  trialed  at  1 1  organizations  within  6  companies.  These  have 
included  military,  commercial,  and  in-house  software  development  organizations.  The 
feedback  from  the  additional  trials  has  tended  to  affirm  the  issues  raised  during  the  Pacific  Bell 
trials.  Nonetheless,  the  general  response  to  the  method  has  been  very  positive. 
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6  Summary 


The  development  of  the  interim  profile  method  was  driven  by  a  need  among  software 
organizations  for  a  way  to  monitor  improvements  in  software  process  maturity  between 
software  process  assessments.  Because  the  method  would  be  employed  frequently,  it  had  to 
be  relatively  low  in  cost  to  conduct  and  require  minimal  disruption  in  the  work  schedule  of  the 
organization.  The  interim  profile  method  was  designed  to  meet  these  requirements.  It  relies 
heavily  on  the  CMM-based  maturity  questionnaire  as  a  means  for  gathering  data  from 
software  projects  on  their  software  process  maturity.  Project  members  typically  spend  less 
than  two  hours  completing  the  questionnaire.  To  aid  their  understanding  of  how  improvement 
efforts  are  progressing,  software  projects  receive  results  based  on  the  MQ  responses  of  the 
project  members.  Organizational  process  improvement  status  is  derived  by  aggregating  the 
project  results.  The  method  satisfies  a  need  expressed  by  the  software  community  and 
expands  the  suite  of  SEI  appraisal  products. 
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Appendix  A  Maturity  Questionnaire  Sampie 


Defect  Prevention  involves  analyzing  defects  that  were  encountered  in  the  past  and  taking  specific 
actions  to  prevent  the  occurrence  of  those  types  of  defects  in  the  future.  The  defects  may  have  been 
identified  on  other  projects  as  well  as  in  earlier  stages  or  tasks  of  the  current  project.  Trends  are 
analyzed  to  track  the  types  of  defects  that  have  been  encountered  and  to  identify  defects  that  are  likely 
to  recur.  Both  the  project  and  the  organization  take  specific  actions  to  prevent  recurrence  of  the 
defects. 

audit  -  An  independent  examination  of  a  work  product  or  set  of  work  products  to  assess  compliance  with  specifications, 
standards,  contractual  agreements,  or  other  criteria. 

causal  analysis  meeting  -  A  meeting,  conducted  after  completing  a  specific  task,  to  analyze  defects  uncovered  during  the 
performance  of  that  task. 

common  cause  (of  a  defect)  -  A  cause  of  a  defect  that  is  inherently  part  of  a  process  or  system.  Common  causes  affect 
every  outcome  of  the  process  and  everyone  working  in  the  process. 

policy  -  A  guiding  principle,  typically  established  by  senior  management,  which  is  adopted  by  an  organization  or  project 
to  influence  and  determine  decisions. 

software  quality  assurance  (SQA)  -  (1)  A  planned  and  systematic  pattern  of  all  actions  necessary  to  provide  adequate 
confidence  that  a  software  work  product  conforms  to  established  technical  requirements.  (2)  A  set  of  activities  designed 
to  evaluate  the  process  by  which  software  work  products  are  developed  and/or  maintained. 


1  Are  defect  prevention  activities  planned? 


□  □  □  □ 


□  □  □  □  □ 


2  Does  the  project  conduct  causal  analysis  meetings 
to  identify  common  causes  of  defects? 


□  □  □  □ 


3  Once  identified,  are  common  causes  of  defects 
prioritized  and  systematically  eliminated? 


□  □  □  □ 
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Appendix  B  Sample  Project  Profile 


Interim  Profile 
Key  Process  Area 
Project  Profile 


Organization:  Division  XYZ 
Project:  999  p- 


May  18, 1993 


Peer  Reviews 
Intergroup  Coord 
Sw  Product  Engr 
Integrated  Sw  Mgt 
Training  Program 
Org  Process  Def 
Org  Process  Focus 


NS  PS  FS 


Maturity  Levei 

Maturity  ievei  is  determined  by 
the  highest  levei  at  which  all 
KPAs  up  to  and  including  that 
level  are  fully  satisfied  (or  not 
applicable). 


Legend 

NS  s  not  satisfied; 
PS  s  partially  satisfied; 

FS  s  fully  satisfied; 
blank  s  not  applicable. 


Configuration  Mgt 
QA 

Subcontract  Mgt 
Project  Tracking 
Project  Planning 
Requirements  Mgt 


48% 

16% 

100% 


NS  PS  FS 


Reliability  Factors 

Overall  degree  of  confidence 
"Yes"  degree  of  confidence  * 
%  of  group  responding:  6/6 
Respondent  experience: 

<  3  years 
>  13  years 


"Respondents'  confidence  in  answering  each 
question  (rating  >s3  on  a  scale  of  1-5) 


Appendix  C  Sample  Organization  Profile 


NS  s  not  satisfied; 
PS  s  partially  satisfied; 

FS  s  fully  satisfied; 
blank  s  not  applicable. 


Configuration  Mgt 
QA 

Subcontract  Mgt  i 
Project  Tracking 
Project  Planning 
Requirements  Mgt 


48% 

16% 

100% 


NS  PS  FS 


Reliability  Factors 

Overall  degree  of  confidence 
"Yes"  degree  of  confidence  * 
%  of  group  responding:  6/6 
Respondent  experience: 

<  3  years 
>  13  years 


"Respondents'  confidence  in  answering  each 
question  (rating  >s3  on  a  scaie  of  1-5) 


Appendix  D  Use  of  Interim  Profile  Results  Survey 

Following  the  initial  distribution  of  profiles,  a  simple  one-page  survey  may  be  used  to  elicit 
feedback  from  the  recipient  of  the  profiles.  Use  of  this  type  of  feedback  allows  the  recipients 
to  focus  their  review  of  the  draft  profiles,  and  also  allows  the  IP  practitioner  to  gage  the 
acceptability  (or  validity)  of  the  results.  The  following  questions  were  used  with  the  first  full 
implementation  of  the  IP  method  at  Pacific  Bell. 

Interim  Profile  Results  Survey 

Please  rate  your  level  of  agreement  with  the  following  statements  on  a  scale  of  1  -  7  where  : 
1  =  Completely  Disagree,  7  =  Completely  Agree 

1.  The  results  on  my  profile  accurately  reflect  the  software  engineering  maturity  status  of  my 

work  group.  1  2  3  4  5  6  7 

2.  An  adequate  number  of  people  responded  to  the  questionnaire  to  represent  the  nature  of 

our  group.  1  2  3  4  5  6  7 

3  The  people  from  my  group  responding  to  tfie  questionnaire  are  knowledgable  about  all 
aspects  of  the  work  performed  by  our  group.  1  2  3  4  5  6  7 

4.  The  questions  on  the  maturity  questionnaire  are  appropriate  to  the  kind  of  work  my  group 

does.  1  2  3  4  5  6  7 

5.  There  were  enough  questions  on  the  maturity  questionnaire  to  adequately  capture  the  key 

process  area  practices  in  my  organization.  1  2  3  4  5  6  7 

COMMENTS 

When  using  this  type  of  feedback  mechanism,  it  is  important  to  have  space  for  open-ended 
comments.  Even  if  the  response  rate  to  the  survey  is  low,  the  feedback  provided  in  the 
comments  can  provide  valuable  insights  for  interpreting  results  and  for  improving  the  IP 
method.  The  ratings  provided  for  the  questions  may  also  be  used  for  informative  ‘post-hoc’ 
analyses  of  the  data  collected  in  the  IP  process.  Examples  of  ‘post-hoc’  analyses  include: 

•  Examine  the  number  of  people  who  participated  for  each  project  and  the 
distribution  of  ratings  given  for  question  2  on  the  survey  (which  reads  “An 
adequate  number  of  people  responded  to  the  questionnaire....”).  If  projects  with 
a  low  turn  out  at  the  MQ  administrations  tend  to  give  poor  ratings  on  the 
question,  this  might  provide  evidence  of  the  need  for  wide  participation  which 
you  can  take  to  your  management. 

•  Examine  the  confidence  data  for  respondents  completing  the  results  survey  to 
see  if  the  answers  provided  for  questions  1 , 3,  and  4  relate  to  the  confidence  of 
the  survey  respondent. 

•  Examine  the  number  of  maturity  questions  that  were  answered  as  “does  not 
apply”  in  light  of  the  responses  to  survey  question  4  (which  reads  “The  questions 
on  the  maturity  questionnaire  are  appropriate  to  the  kind  of  work  my  group 
does”). 
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